Product testing requires a large number of tests to be run, and each test result must be stored. For example, one software product in development can require a set of 16,500 tests to be regularly run to ensure proper operation. These tests can take 15 hours to run using one test machine. If a developer makes a code change there is the distinct possibility that one or more of the 16,500 tests will regress. Experience has shown that if the developers do not run the tests over their changed code on one day, they are highly likely to have a significant number of regressions to deal with the next day, and product development and release schedules will suffer.
If a test regresses it is the responsibility of a test engineer to determine what code change has caused the regression. This involves running the failing test over recompiled copies of the product, where each copy has had one or more of the code changes removed, to see if this causes the test to pass. If a code change is “backed off”, and the test subsequently passes, the code causing the regression has been found, and the relevant developer can be notified in order to fix it.
Typically the above process can take up to half an hour per regression due to the number of recompilations required, so if there are many regressions the assigned test engineer can spend a whole day resolving the regressions caused during the previous night's test run.
There is, therefore, a need in the art for a system, process and computer program product for efficiently and automatically identifying the cause of code regressions.